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We asked the same 60 questions using DRS (digital reference services) in Japanese public libraries, face-to-face 
reference services and Q & A (question and answer) sites. It was found that: (1) The correct answer ratio of DRS is 
higher than that of Q & A sites; (2) DRS takes longer to provide answers as compared to Q & A sites; and (3) The 
correct answer ratio of face-to-face reference services in public libraries that provide (do not provide) DRS is higher 
than (the same as) that of Q & A sites. Considering that a majority of the Japanese public libraries do not provide 
DRS, the result indicates that Q & A sites and face-to-face reference services are comparable in terms of their ability 
to answer questions. However, the public libraries that earnestly provide DRS can outperform Q & A sites. 
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Introduction 

Reference services in Japanese public libraries are now being challenged by Q & A (question and answer) 
sites on the Web. In this paper, we define Q & A sites as those sites where anybody can ask questions for free, 
and in many cases, receive answers from other users or specialists. In Japan, “Yahoo! Chiebukuro” and 
“Oshiete! goo” are typical Q & A sites. Employing unobtrusive testing, Tsuji, Umehara, Kikawada and Hara 
(2010) asked the same questions using Oshiete! goo and face-to-face reference services (henceforth 
“face-to-face”) in Japanese public libraries and found no difference between the accuracy of answers provided 
by both these sources. However, Tsuji et al. (2010) investigated only face-to-face and municipal libraries. If 
they had investigated e-mail DRS (digital reference services) and prefectural libraries, which possess more 
human and information resources, the result may be varied. DRS are similar to Q & A sites in terms of their 
availability on the Internet. If the answers obtained by using DRS prove to be more accurate than those from Q 
& A sites, people may begin using DRS instead of Q & A sites. Against this background, we asked the same 60 
questions to: (1) DRS in prefectural and municipal libraries; (2) face-to-face; and (3) Q & A sites. Subsequently, 
we compared the accuracy of the answers provided by each of these sources. 

This paper may be criticized with respect to the following two problems: (1) unobtrusive testing; and (2) 
focusing on the correct answer ratio. With regard to “unobtrusive testing”, it is believed that unobtrusive testing 
may lower the morale of librarians who are investigated and may waste library resources (Weech, 1974). 
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However, providing feedback to the libraries based on findings may aid them in improving their services 
(Kaske & Arnold, 2002). In addition, in the US, where progressive reference services are observed, unobtrusive 
testing is performed rather frequently, as mentioned subsequently. With regard to “focusing on the correct 
answer ratio”, we understand that reference services consist of various services and therefore, various 
evaluation measures have been proposed. A popular measure is the degree of user satisfaction, which provides 
potential for future research. 

Related Studies 

In this section, we describe the related studies and current status of Q & A sites and reference services. 

Q & A Sites 

There are numerous Q & A sites in Japan, including Yahoo! Chiebukuro, Oshiete! goo, OKWave, 
Hatena:: Question and livedoor knowledge 1 . Among them, Yahoo! Chiebukuro is the largest and comprises 
over 60.5 million Q & A and had 8.0 million registered users as of April 2011. Oshiete! goo and OKWave are 
the second largest and both comprise 5.5 million Q & A each. In the US, Yahoo! Answers, WikiAnswers and 
AnswerBag are the most popular 2 . According to Hitwise (2008), the number of Q & A users in the US is 
rapidly increasing and number of visitors to Q & A sites increased by 889% from February 2006 to February 
2008. Coffman and Arret (2004) stated that a large number of commercial reference services that worried 
librarians had either died, or were so gravely wounded that they could no longer constitute a threat to anybody. 
However, currently, Q & A sites are gaining popularity, and we are uncertain whether or not their optimism 
will hold in the future. 

There are no prior studies evaluating the accuracy of answers obtained from Japanese Q & A sites except 
Tsuji et al. (2010), whose work has been previously mentioned. As for the US and the UK, Margariti and 
Chowdhury (2003), Lochore (2004), Roush (2006) and Bivens-Tatum (2001) evaluated the Q & A sites. 
However, their investigations were relatively limited. 

Reference Services 

Crews (1988) reviewed 39 papers, which employed unobtrusive testing for evaluating the accuracy of 
answers provided by reference services. Among them, Hernon and McClure (1986) and Crowley (1971) are the 
most popular. Kaske and Arnold (2002), Lochore (2004) and Nilsen and Ross (2006) investigated the accuracy 
of answers provided by DRS. Lochore (2004) asked ten questions using DRS (a librarian in the UK and DRS 
by UCLA (University of California, Los Angeles)) and a Q & A site (AllExperts) and found that the two DRS 
provided three and five correct answers, respectively, and the Q & A site provided four. Studies comparing the 
accuracy of answers obtained by using DRS and Q & A sites are limited. 

Method 

In this section, we will describe: (1) the public libraries and Q & A sites that we used for the investigation; 
(2) questions; and (3) answer categories. 



1 Yahoo! Chiebukuro (Retrieved from http://chiebukuro.yahoo.co.jp/); Oshiete! Goo (Retrieved from http://oshiete.goo.ne.jp/); 
OKWave (Retrieved from http://okwave.jp/); Hatena: :Question (Retrieved from http://q.hatena.ne.jp/); livedoor knowledge 
(Retrieved from http://knowledge.livedoor.com/). 

2 Yahoo! Answers (Retrieved from http://answers.yahoo.com/); WikiAnswers (Retrieved from http://wiki.answers.com/); 
AnswerBag (Retrieved from http://www.answerbag.com/). 
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Public Libraries and Q & A Sites 

We selected six prefectural libraries in Kanto district and 25 municipal libraries around Tokyo and asked 
questions face-to-face. In addition, we chose six prefectural libraries and three municipal libraries and asked 
questions using their DRS. The number of samples of libraries for evaluating the accuracy of answers provided 
by DRS is small because Japanese municipal libraries do not provide DRS, and even if they do, the users are 
restricted to the residents of their service area for a majority of such libraries. With respect to Q & A sites, we 
selected Y ahoo ! Chiebukuro and Oshiete ! goo. 

Questions 

For the purpose of our study, questions must be answerable and easy for us to evaluate the accuracy of 
their answers. Based on this, we employed 30 questions that have been used in the classes of “reference service 
practice” at our university and 30 questions that have been asked by users in public libraries and stored in their 
reference records. It must be noted that confidential questions asked by users were not included. In 2009, we 
asked the 60 above-mentioned questions using: (1) face-to-face (mainly from August 28 to October 6); (2) DRS 
(from November 17 to December 2); (3) Q & A sites (from December 9 to December 1 1). 

With respect to face-to-face, on an average, we asked two questions to each library. With respect to DRS, 
we asked about eight and six questions to each prefectural and municipal library respectively. It was ensured 
that the same questions were not asked using face-to-face and DRS in the same libraries. With regard to Q & A 
sites, we asked a different set of 30 questions each using Yahoo! Chiebukuro and Oshiete! goo. 

Answer Categories 

We classified the answers based on: (1) amount of correct answers; (2) amount of incorrect answers; and 
(3) whether or not the correct answer was provided directly (i.e., by introducing certain books, etc.). A few of 
the questions that we employed consisted of multiple sub-questions. For example, a question that enquires the 
title and publisher of a book consists of two sub-questions, one each for enquiring about the title and publisher 
of the book. Therefore, the answer to one question can be partially correct (for example, the title could be 
correct and the name of the publisher could be incorrect or vice versa). On the basis of these classifications, the 
answers provided by the three services were classified into nine categories in the following manner: (1) Correct 
answer was provided directly and completely. Incorrect answer was not provided; (2) Books, Web pages or 
other materials that included the correct answer were indicated; the answer could be found easily. Incorrect 
answer was not provided; (3) Books, Web pages or other materials that included the correct answer were 
indicated; however, it was not easy to find the answer (i.e., it required additional time, skill or knowledge). 
Incorrect answer was not provided; (4) Books, Web pages or other materials that included the correct answer 
were indicated; however, it was not easy to find the answer. Incorrect answer was provided; (5) Paid of the 
correct answer was provided directly. Incorrect answer was not provided; (6) Pail of the correct answer was 
provided directly. Incorrect answer was provided; (7) Neither the correct answer nor reference materials were 
provided (this refers to those cases where librarians were unable to find the answer or Q & A sites provided no 
answer). Incorrect answer was not provided; (8) Neither the correct answer nor reference materials were 
provided. Incorrect answer was provided; and (9) Difficulties to evaluate the accuracy of the answer (includes 
cases where the answer advises visiting the national library or provides names of a large number of books and 
advises referring to all of them). 

Henceforth, we term the ratio of answer categories A and B among all the answers as “correct answer 
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ratio”. In addition, we group answer categories A and B to represent “Good”; C and D to represent “Not Good”; 
and E, F, G and H to represent “Bad” in the interest of brevity. 

Results and Discussions 

In this section, we initially indicate the basic results, followed by the results according to question topics, 
time required, answers sources and relationship between DRS and face-to-face. 

Basic Results 

The correct answer ratios of DRS, face-to-face and Q & A sites are indicated in Tables 1, 2 and 3 
respectively. As indicated in Tables 1 and 3, 20 out of 46 (i.e., 43.5%) answers provided by DRS were 
classified into category A while only 8 out of 30 answers (i.e., 26.7%) provided by Yahoo! Chiebukuro were 
classified into category A. 

Table 1 



Answer Categories by DRS 







Prefectural (%) 




Municipal (%) 


A 


20 


(43.5) 


8 


(57.1) 


B 


11 


(23.9) 


4 


(28.6) 


C 


9 


(19.6) 


1 


(7.1) 


D 


0 


(0.0) 


0 


(0.0) 


E 


2 


(4.3) 


0 


(0.0) 


F 


1 


(2.2) 


0 


(0.0) 


G 


1 


(2.2) 


1 


(7.1) 


H 


0 


(0.0) 


0 


(0.0) 


I 


2 


(4.3) 


0 


(0.0) 


Total 


46 


(100) 


14 


(100) 


Table 2 










Answer Categories by Face-to-Face 












Prefectural (%) 




Municipal (%) 


A 


6 


(50.0) 


25 


(52.1) 


B 


2 


(16.7) 


5 


(10.4) 


C 


2 


(16.7) 


3 


(6.3) 


D 


1 


(8.3) 


1 


(2.1) 


E 


1 


(8.3) 


6 


(12.5) 


F 


0 


(0.0) 


0 


(0.0) 


G 


0 


(0.0) 


5 


(10.4) 


H 


0 


(0.0) 


3 


(6.3) 


I 


0 


(0.0) 


0 


(0.0) 


Total 


12 


(100) 


48 


(100) 



Based on Tables 1, 2 and 3, the following can be established: (1) The correct answer ratio of DRS in 
municipal libraries is 85.7% (i.e., 57.1% + 28.6%), which is higher than that of prefectural libraries (67.4%, i.e., 
43.5% + 23.9%), as well as the correct answer ratio of face-to-face and Q & A sites; (2) The correct answer ratio 
of DRS in a prefectural library is higher than that from a Q & A site; (3) The number of incorrect answers 
provided by DRS (i.e., answers categories D, F and H) are fewer than those by face-to-face and Q & A sites; (4) 
The correct answer ratios of face-to-face in prefectural and municipal libraries do not differ significantly; (5) The 
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correct answer ratio of face-to-face in municipal libraries is 62.5% (i.e., 52.1% + 10.4%), which is slightly 
higher than that of Oshiete! goo (56.6%, i.e., 43.3% + 13.3%). Tsuji et al. (2010) have described that the correct 
answer ratios of face-to-face in municipal libraries and that of Oshiete! goo do not differ significantly. Similar 
results have been obtained in this study; and (6) The correct answer ratio of Oshiete! goo (56.6%) is a slightly 
higher than that of Yahoo! Chiebukuro (50.0%, i.e., 26.7% + 23.3%), although it is not statistically significant. 

Table 3 



Answer Categories by Q & A Sites 





Yahoo! Chiebukuro (%) 




Oshiete! Goo (%) 


A 


8 


(26.7) 


13 


(43.3) 


B 


7 


(23.3) 


4 


(13.3) 


C 


3 


(10.0) 


2 


(6.7) 


D 


0 


(0.0) 


1 


(3.3) 


E 


3 


(10.0) 


3 


(10.0) 


F 


0 


(0.0) 


0 


(0.0) 


G 


5 


(16.7) 


6 


(20.0) 


H 


2 


(6.7) 


1 


(3.3) 


I 


2 


(6.7) 


0 


(0.0) 


Total 


30 


(100) 


30 


(100) 



Question Topics 

We classified the questions into eight topics (persons/organizations, geography, history, events, languages, 
books, reference books and joumals/newspapers) and investigated the correct answer ratio for each topic. Tsuji 
et al. (2010) reported that the correct answer ratio of Q & A sites for questions with regard to “books” was 
higher than that of face-to-face. However, such tendencies were not observed in our study. Correct answer ratios 
of face-to-face for questions regarding “books” were 60.0% and 55.6% for prefectural and municipal libraries 
respectively. On the other hand, the correct answer ratios for questions regarding “books” were 50.0% and 
46.7% of Yahoo! Chiebukuro and Oshiete! goo respectively. With respect to DRS, the correct answer ratios for 
questions regarding “books” were 75.0% and 85.7% for prefectural and municipal libraries respectively. 

Time Required 

We investigated the time required to obtain answers from DRS, face-to-face and Q & A sites and their 
respective correct answer ratios. In this paper, “time required to obtain answers” is defined as the sum of “the 
time taken by the services to reply” and “time taken by us to find answers by referring materials, such as books 
and Web pages, which were suggested in the replies”. Undoubtedly, the time taken in the latter case is nil when 
the replies directly indicated the answers (i.e., in case of category A answers). 

The time required to obtain answers from DRS, face-to-face and Q & A sites and their respective correct 
answer ratios are shown in Tables 4, 5 and 6 respectively. As indicated in Table 4, nine answers were provided 
by DRS in prefectural libraries after one day and within two days and 77.8% (i.e., seven answers) of them were 
classified into category A or B. We can see in these tables that we obtained answers for only two questions 
within an hour using DRS. On the other hand, we obtained answers for 53 and 22 questions within an hour 
using face-to-face and Q & A sites respectively. It may be the future task for DRS librarians to reduce the time 
to answer questions. However, we must keep in mind that the correct answer ratio of DRS was the highest 
among these services. 
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Table 4 

Time Required by DRS 







Prefectural (%) 




Municipal (%) 


Within 1 hour 


2 


(50.0) 


0 


(-) 


1 to 2 hours 


1 


(100.0) 


3 


(66.7) 


2 to 3 hours 


0 


(-) 


2 


(100.0) 


3 to 4 hours 


5 


(60.0) 


1 


(100.0) 


4 to 5 hours 


5 


(60.0) 


0 


(-) 


5 to 6 hours 


4 


(75.0) 


2 


(100.0) 


6 hours to 1 day 


5 


(80.0) 


1 


(100.0) 


1 to 2 days 


9 


(77.8) 


2 


(-) 


2 to 3 days 


2 


(50.0) 


1 


(-) 


3 to 7 days 


8 


(62.5) 


0 


(-) 


More than 7 days 


4 


(75.0) 


0 


(-) 


Other 


1 


(0.0) 


2 


(100.0) 


Table 5 










Time Required by Face-to-Face 












Prefectural (%) 




Municipal (%) 


Within 5 minutes 


0 


(-) 


5 


(80.0) 


5 to 10 minutes 


1 


(100.0) 


8 


(37.5) 


10 to 15 minutes 


1 


(100.0) 


9 


(55.6) 


15 to 20 minutes 


0 


(-) 


1 


(0.0) 


20 to 30 minutes 


4 


(25.0) 


6 


(66.7) 


30 minutes to 1 hour 


3 


(100.0) 


15 


(73.3) 


1 to 2 hours 


1 


(0.0) 


0 


(-) 


More than 2 hours 


0 


(-) 


0 


(-) 


Other 


2 


(100.0) 


4 


(75.0) 


Table 6 










Time Required by Q & A Sites 












Yahoo! Chiebukuro (%) 




Oshiete!goo (%) 


Within 5 minutes 


2 


(100.0) 


2 


(50.0) 


5 to 10 minutes 


1 


(100.0) 


4 


(50.0) 


10 to 15 minutes 


3 


(66.7) 


3 


(100.0) 


15 to 20 minutes 


3 


(66.7) 


0 


(-) 


20 to 30 minutes 


2 


(100.0) 


0 


(-) 


30 minutes to 1 hour 


1 


(100.0) 


3 


(33.0) 


1 to 2 hours 


3 


(66.7) 


5 


(80.0) 


2 to 3 hours 


0 


(-) 


1 


(100.0) 


3 to 4 hours 


1 


(0.0) 


2 


(50.0) 


4 to 6 hours 


1 


(0.0) 


0 


(-) 


6 hours to 1 day 


5 


(60.0) 


4 


(25.0) 


1 to 2 days 


0 


(-) 


3 


(66.7) 


2 to 3 days 


3 


(0.0) 


2 


(100.0) 


No answer 


5 


(0.0) 


3 


(0.0) 
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Answers were obtained for 29 and 1 1 questions within 30 minutes using face-to-face in municipal libraries 
and Yahoo! Chiebukuro respectively. Concerning these, the average correct answer ratios of face-to-face in 
municipal libraries and Yahoo! Chiebukuro which can be calculated from those in Tables 5 and 6 are 55.2% 
and 81.8%, respectively (for instance, the average correct answer ratio of Yahoo! Chiebukuro is (2 + 1 + 2 + 2 
+ l)/(2 + 1 + 3 + 3+ 2) = 81.8%). The face-to-face is the fastest source for obtaining answers, and Q & A sites 
are comparatively faster than DRS . 

Sources 

We investigated the sources that were referred in the answers obtained from DRS, face-to-face and Q & A 
sites. We cannot indicate all the sources referred by the other services owing to the limitation of space. In brief, a 
majority of the referred sources in DRS were dictionaries (for 20 questions) followed by NDL-OPAC 3 . DRS in 
prefectural and municipal libraries often did not refer sources (for six and four questions in prefectural and 
municipal libraries respectively. Five of these answers were for questions that investigated titles or publishers of 
books 4 ). If the source (such as NDL-OPAC (Online Public Access Catalog of National Diet Library), Webcat 
Plus 5 or their own OP AC) is indicated, the users could leam of its existence and may use it subsequently 6 . 

Q & A sites often referred no source as well (for 17 questions) and referred Wikipedia (for nine questions). 
However, they referred NDL-OPAC or Webcat Plus (for nine questions) as sources including DRS and reliable 
government homepages (for five questions). Furthermore, Q & A sites referred the image of the body of a book 
provided by Google Books (for one question). Tsuji et al. (2010) also found that a few of the answers on Q & A 
sites referred such images of books provided by Google Books. If we regard printed books as reliable sources, 
it may be established that Q & A sites as well as reference services in libraries refer reliable sources for their 
answers. In addition, reference librarians must learn to utilize such sources for their DRS. 

Face-to-Face Where DRS Is Being Provided 

Table 7 indicates the correct answer ratios of face-to-face by three categories of public libraries including: 
(1) libraries that provide DRS; (2) libraries whose central library provides DRS (“C -Provide” in Table 7); and 
(3) libraries that do not provide DRS. Table 7 indicates the correct answer ratio of face-to-face in public 
libraries that provide DRS is 87.5% (i.e., 7/(7 + 0 + 1)) and is significantly better than those that do not provide 
DRS (57.7%, i.e., 15/(15 + 3 + 8)). Tables 2 and 3 indicate that the correct answer ratios of face-to-face by 
public libraries that provide DRS outperform those of Q & A sites. However, those public libraries that do not 
provide DRS do not differ significantly from the correct answer ratios of Q & A sites. 

Table 7 



Correct Answer Ratio of Face-to-Face by Three Categories of Public Libraries 







Good (%) 




No good (%) 




Bad (%) 


Provide 


7 


(87.5) 


0 


(0.0) 


1 


(12.5) 


C-provide 


8 


(57.1) 


1 


(7.1) 


5 


(35.7) 


Not provide 


15 


(57.0) 


3 


(11.5) 


8 


(30.8) 



NDL-OPAC is an OPAC in Japan. 

4 Five other answers were for those questions that the librarians could not find answers to. 

5 Webcat Plus is a nation-wide OPAC of Japanese university libraries. 

6 With regard to face-to-face, no source was referred for only four questions. 
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Conclusions 

We asked the same 60 questions using DRS in Japanese public libraries, face-to-face, and Q & A sites. It 
was found that: (1) The correct answer ratio of DRS in prefectural libraries is higher than that of Q & A sites; 
(2) The correct answer ratio of DRS in municipal libraries is rather high; (3) DRS takes longer to provide 
answers to questions as compared to Q & A sites; (4) There is no significant difference between the correct 
answer ratio of face-to-face in public libraries that do not provide DRS and that of Q & A sites; and (5) The 
correct answer ratio of face-to-face in public libraries that provide DRS is higher than that of Q & A sites. 

Considering that most of the Japanese public libraries do not provide DRS, the result indicates that Q & A 
sites and ordinary face-to-face reference are comparable in terms of their ability to answer questions. However, 
if the public libraries earnestly provide DRS, they could outperform Q & A sites. We should take these results 
into account while considering the future of reference services. 

Subsequently, we will investigate DRS and face-to-face in locations other than Kanto district. In addition, 
we will interview DRS librarians in order to examine the relationship between the process of answering 
questions in libraries and correct answer ratio. Finally, we would like to evaluate the satisfaction levels of users 
employing DRS. 
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